state bulk process increment analyt work address state dataflow program rapidli sift huge evolv data set data intens applic perform complex multi step comput success gener data inflow weekli web crawl daili imag video upload log file grow social network programm simpli run entir dataflow data arriv grossli ineffici increas result latenc squander hardwar resourc energi altern programm prior result increment incorpor current larg scale data process tool map reduc dryad limit programm incorpor state data parallel program straightforward approach incorpor state result custom fragil code disappoint perform work present gener architectur continu bulk process cbp rais level abstract build increment applic core flexibl groupwis process oper take state explicit input unifi state program data parallel oper afford fundament opportun minim movement data underli process system case studi show small set flexibl dataflow primit perform web analyt mine larg scale evolv graph increment fashion experi prototyp real world data signific data movement run time reduct rel current practic increment comput pagerank cbp reduc data movement cut run time half web crawl interact analysi web dscale data support interact queri webscal data basic approach view queri phase activ suppli queri templat suppli specif instanti templat interact respons offer phase instanc problem studi past olap web search pursu gener formul aim build gener phase queri system gener data dataflow program develop data centric program user run portion program real data behav output make easier formul understand compos program correctli compar examin program logic larg input data set experiment run time consum ineffici sampl input data work select oper filter join lead empti result sampl input index present gener bias sampl effici method need gener input data data centric program focu import categori data centric program dataflow program illustr displai seri intermedi data tabl occur pair oper introduc studi problem gener intermedi data dataflow program manner illustr semant oper keep data small identifi major obstacl imped naiv approach highli select oper noninvert oper offer techniqu deal obstacl techniqu perform real dataflow program yahoo web analyt build highlevel dataflow system top mapreduc pig experi increasingli organ captur transform analyz enorm data set promin exampl includ internet compani scienc map reduc scalabl dataflow paradigm popular applic simpl explicit dataflow program model favor tradit high level declar approach sql hand extrem simplic map reduc lead low level hack deal step branch dataflow aris practic user repeatedli code standard oper join hand practic wast time introduc bug harm readabl imped optim pig high level dataflow system aim sweet spot sql map reduc pig offer sql style high level data manipul construct assembl explicit dataflow interleav custom map reduc style function execut pig program compil sequenc map reduc job execut hadoop map reduc environ pig hadoop open sourc project administ apach softwar foundat paper describ challeng face develop pig report perform comparison pig execut raw map reduc execut parallel evalu composit aggreg queri abstract aggreg measur summar subset data valuabl exploratori analysi decis support depend aggreg easili comput class queri call composit subset measur previous introduc correl aggreg queri easili express paper consid evalu composit subset measur queri larg distribut system describ cross node data redistribut strategi take account nest structur queri main idea group data block cube space aggreg gener local block leverag previous propos optim block partit scheme overlap block slide window aggreg handl guarante final answer union local result duplic expens data combin step identifi import partit paramet propos optim algorithm demonstr effect optim minim queri respons time parallel evalu composit aggreg queri abstract aggreg measur summar subset data valuabl exploratori analysi decis support depend aggreg easili comput class queri call composit subset measur previous introduc correl aggreg queri easili express paper consid evalu composit subset measur queri larg distribut system describ cross node data redistribut strategi take account nest structur queri main idea group data block cube space aggreg gener local block leverag previous propos optim block partit scheme overlap block slide window aggreg handl guarante final answer union local result duplic expens data combin step identifi import partit paramet propos optim algorithm demonstr effect optim minim queri respons time pig latin dso dforeign languag data process grow ad hoc analysi extrem larg data set internet compani innov critic depend analyz terabyt data collect dai parallel databas product teradata offer solut prohibit expens scale peopl analyz data entrench procedur programm find declar sql style unnatur success procedur map reduc program model scalabl implement commod hardwar evid map reduc paradigm low level rigid lead great deal custom user code hard maintain reus describ languag call pig latin design fit sweet spot declar style sql low level procedur style map reduc accompani system pig fulli implement compil pig latin physic plan execut hadoop open sourc map reduc implement give exampl engin yahoo pig dramat reduc time requir develop execut data analysi task compar hadoop directli report debug environ integr pig lead higher product gain pig open sourc apach incub project gener automat optim parallel dataflow program larg scale parallel dataflow system dryad map reduc attract signific attent recent high level dataflow languag pig latin sawzal layer top system enabl faster program develop maintain code languag engend greater transpar program structur open opportun automat optim paper propos set optim strategi context draw extend techniqu databas commun crawl order search impact studi priorit fetch page object maxim qualiti search result object fetch page impact impact page equal number time page appear top search result queri constant impact page depend relev score queri turn depend page content main difficulti li estim impact page fetch impact estim base limit inform prior fetch page content url string number link refer anchortext formal character problem studi hard leverag formal design impactdriven crawl polici demonstr effect real world data techniqu ensur crawler acquir content relev tail topic obscur interest user redundantli accumul content popular topic recrawl schedul base inform longev crucial web crawler distinguish ephemer persist content ephemer content quot dai worth crawl time reach index longer repres web page acquir hand content persist multipl page updat recent blog post worth acquir match page true content sustain period time paper character longev inform found web empir measur gener model coincid measur develop recrawl schedul polici longev account show experi real web data polici obtain fresh lower cost compar previou approach scalabl queri result cach web applic backend databas system perform bottleneck run web applic common approach scale databas compon queri result cach face challeng maintain high cach hit rate effici ensur cach consist databas updat paper introduc ferdinand proxi base cooper queri result cach fulli distribut consist manag maintain high cach hit rate ferdinand local queri result cach proxi server distribut cach consist manag implement highli scalabl publish subscrib system implement fulli function ferdinand prototyp evalu perform compar altern queri cach approach show high cach hit rate consist manag critic ferdinand perform gain exist system relax text search taxonomi paper propos document retriev model text queri augment multi dimension taxonomi restrict restrict relax cost result qualiti model applic arena includ multifacet product local search document augment hierarch metadata topic locat present effici algorithm index queri process retriev model decompos queri process problem onlin search problem determin correct level relax cost incur gener top result budget relax search problem result relax cost produc minim cost show problem solvabl hierarch dimens np hard dimens admit effici approxim algorithm provabl guarante present experiment result evalu algorithm synthet real data show order magnitud improv baselin algorithm schedul share scan larg data file studi schedul scan larg data file presenc simultan request common set file object maxim rate process file share scan file aggress impos undu wait time individu job schedul problem aris batch data process environ map reduc system handl ten thousand process request daili share set file demonstr convent schedul techniqu shortest job perform presenc cross job share opportun deriv famili schedul polici specif target sharabl workload schedul polici revolv notion equal good schedul nonshar scan ahead share io work futur job arriv rate sharabl futur job expect high evalu polici simul vari synthet real workload demonstr signific perform gain compar convent schedul approach invalid clue databas scalabl servic scalabl data intens web applic databas scalabl servic dbss cach applic queri result answer queri behalf applic address secur privaci concern dbss encrypt data pass dbss dbss invalid larg region cach data updat occur invalid precis dbss order result invalid inevit reveal properti data paper present invalid clue gener techniqu enabl applic reveal data dbss limit number unnecessari invalid compar previou approach invalid clue provid applic significantli improv tradeoff secur privaci scalabl experi web applic benchmark prototyp dbss built confirm invalid clue low overhead effect gener techniqu applic balanc privaci scalabl configur model distribut data storag navigationaid retriev user search inform hypermedia environ perform queri manual navig convent text hypertext retriev paradigm post queri navig account paper propos retriev paradigm call navig aid retriev nar treat queri navig class activ nar paradigm queri mean identifi start point navig navig guid base inform suppli queri nar gener convent probabilist inform retriev paradigm implicitli assum navig take place paper discover web comput shortest path uncertainti problem estim length shortest path dag edg length approxim determin cost obtain posit neg result exact problem approxim handl advertis unknown qualiti search advertis simultan scalabl secur data dintens web applic web applic databas compon bottleneck scalabl provid parti databas scalabl servic provid dssp cach applic data suppli queri answer behalf applic cost effect dssp cach data applic inevit rais concern secur data pass dssp encrypt enhanc secur data updat trigger invalid larg region cach achiev good scalabl virtual imposs tradeoff secur scalabl requir care consider paper studi secur scalabl tradeoff formal empir begin provid method static identifi segment databas encrypt impact scalabl experi prototyp dssp system show effect static analysi method realist benchmark applic studi method enabl signific fraction databas encrypt impact scalabl data encrypt impact scalabl type applic design encrypt thing equal base static analysi method propos scalabl consciou secur design methodolog featur compulsori encrypt highli sensit data credit card inform encrypt data encrypt impair scalabl result secur scalabl tradeoff consid data encrypt impact scalabl greatli simplifi task manag tradeoff scalabl servic dynam web applic find recent frequent item distribut data stream problem maintain frequenc count item occur frequent union multipl distribut data stream na ve method combin approxim frequenc count multipl node tend result excess larg data structur costli transfer node minim commun requir degre precis maintain node count item frequenc manag carefulli introduc concept precis gradient manag precis node arrang hierarch commun structur studi optim problem set precis gradient minim commun provid optim solut minim worst case commun load input introduc variant design perform practic input data conform worst case characterist verifi effect approach empir real world data show method incur substanti commun na ve approach provid error guarante answer shuffl stack deck case partial random rank search engin result degre pagerank number visit measur web page popular significantli influenc rank search result modern search engin assumpt popular close correl qualiti elus concept difficult measur directli correl popular qualiti weak newli creat page receiv visit link wors discoveri content larg queri search engin user focu attent top result newli creat high qualiti page effect shut long time popular propos simpl eleg solut problem introduct control amount random search result rank method offer page chanc prove worth random degrad result qualiti annul benefit achiev tradeoff explor estim qualiti page exploit page high qualiti studi tradeoff analyt simul context econom object function base aggreg result qualiti amort time show modest amount random lead improv search result shuffl stack deck case partial random rank search engin result degre pagerank number visit measur web page popular significantli influenc rank search result modern search engin assumpt popular close correl qualiti elus concept difficult measur directli correl popular qualiti weak newli creat page receiv visit link wors discoveri content larg queri search engin user focu attent top result newli creat high qualiti page effect shut long time popular propos simpl eleg solut problem introduct control amount random search result rank method offer page chanc prove worth random degrad result qualiti annul benefit achiev tradeoff explor estim qualiti page exploit page high qualiti studi tradeoff analyt simul context econom object function base aggreg result qualiti amort time show modest amount random lead improv search result user dcentric web crawl search engin primari gatewai inform access web todai scene search engin crawl web popul local index repositori web page answer user search queri aggreg sens web dynam caus repositori web page date time turn queri answer qualiti degrad consider size dynam degre autonomi web feasibl search engin maintain repositori synchron web paper studi schedul web page select download search engin repositori schedul object maxim qualiti user experi queri search engin begin quantit character discrep content repositori current content live web impact qualiti user experi character lead usercentr metric qualiti search engin local repositori metric deriv polici schedul web page download driven search engin usag free exterior tune paramet focu import subproblem schedul refresh web page present repositori show comput prioriti effici provid extens empir comparison user centric method prior web page refresh strategi real web data result demonstr method requir fewer resourc maintain search engin qualiti level user leav substanti resourc incorpor web page search repositori shuffl stack deck case partial random rank search engin result degre pagerank number visit measur web page popular significantli influenc rank search result modern search engin assumpt popular close correl qualiti elus concept difficult measur directli correl popular qualiti weak newli creat page receiv visit link wors discoveri content larg queri search engin user focu attent top result newli creat high qualiti page effect shut long time popular propos simpl eleg solut problem introduct control amount random search result rank method offer page chanc prove worth random degrad result qualiti annul benefit achiev tradeoff explor estim qualiti page exploit page high qualiti studi tradeoff analyt simul context econom object function base aggreg result qualiti amort time show modest amount random lead improv search result effici monitor queri distribut increasingli common applic data resid multipl dispar locat applic requir central access data simpl solut replic relev data central point forward updat master copi replica special process filter scheme maintain date central data incur signfic commun overhead data highli dynam volum updat larg commun resourc preciou commun reduc priorit filter updat insid network data sourc updat drop replica approxim exact fortun real world applic involv distribut dynam data toler approxim data valu extent approxim replic import techniqu balanc replica precis commun resourc achiev wic gener dpurpos algorithm monitor web inform sourc web univers inform dissemin medium due number factor includ support content dynam grow number web inform provid post real time updat domain auction stock market bulletin board new weather roadwai condit sport score web evolut web search engin perspect seek gain improv insight web search engin cope evolv web attempt provid user date result purpos collect weekli snapshot web site year measur evolut content link structur measur focu aspect potenti interest search engin design evolut link structur time rate creation page distinct content web rate chang content exist page search centric measur degre chang queri process hardwar sensor node combin physic sensor actuat embed processor commun compon advanc significantli decad made larg scale deploy sensor realiti applic rang monitor applic inventori mainten health care militari applic distribut top dk monitor queri analysi data stream topic recent interest motiv applic field network web usag analysi sensor instrument telecommun applic involv monitor answer continu queri data stream produc physic distribut locat previou approach requir stream transmit singl locat central process continu transmiss larg number rapid data stream central locat impract expens studi class queri continu report largest valu obtain distribut data stream top monitor queri interest reduc overhead incur run type monitor queri show transmit entir data stream unnecessari support queri present altern approach reduc commun significantli approach arithmet constraint maintain remot stream sourc ensur recent provid top answer remain valid userspecifi error toler distribut commun occas constraint violat show empir extens simul real world data approach reduc commun cost order magnitud compar altern offer error guarante adapt filter continu queri distribut data stream environ distribut data sourc continu stream updat central processor monitor continu queri distribut data signific commun overhead incur presenc rapid updat stream propos techniqu reduc overhead user regist continu queri precis requir central stream processor instal filter remot data sourc filter adapt chang condit minim stream rate guarante continu queri receiv updat provid answer adequ precis time approach enabl applic trade precis commun overhead fine granular individu adjust precis constraint continu queri stream multi queri workload scenttrail integr brows search web scenttrail integr brows search web comput median uncertainti visual data bound uncertainti visual power facilit data analysi crucial visual system explicitli convei presenc natur degre uncertainti user danger data fals interpret potenti lead inaccur conclus common method denot uncertainti error bar similar techniqu design convei degre statist uncertainti uncertainti model statist form uncertainti bound uncertainti aris properti statist uncertainti deffort cach synchron sourc cooper environ exact synchron sourc data object cach copi achiev due bandwidth resourc constraint stale date copi permit desir minim diverg sourc object cach copi select refresh modifi object call onlin process select object refresh order minim diverg effort synchron approach effort synchron cach coordin process select object refresh paper propos effort synchron schedul polici exploit cooper data sourc cach propos implement polici incur low commun overhead environ larg number sourc algorithm adapt wide fluctuat resourc data updat rate experiment simul synthet real world data demonstr effect algorithm quantifi signific decreas diverg achiev sourc cooper adapt precis set cach approxim valu cach approxim valu exact valu present opportun perform gain exchang decreas precis maxim perform improv cach approxim precis approxim precis easili invalid requir frequent refresh overli imprecis approxim useless applic bypass cach present parameter algorithm adjust precis cach approxim adapt achiev perform data valu precis requir workload vari interv approxim numer valu idea extend kind data approxim algorithm strictli gener previou adapt cach algorithm exact copi set paramet requir approxim exact case algorithm dynam choos cach data implement algorithm test synthet real world data number experiment result report show effect algorithm maxim perform show special case exact cach algorithm perform previou algorithm case bound imprecis accept algorithm easili outperform previou algorithm exact cach datasplash direct manipul environ program semant zoom visual tabular data portal behav comput median uncertainti offer precis dperform tradeoff aggreg queri replic data strict consist replic data infeas requir distribut applic current system permit stale replic inwhich cach copi data valu allow date queri cach data return answer quickli stale answer unboundedli imprecis altern queri remot master data return precis answer potenti poor perform bridg gap extrem propos class replic system call trapp tradeoff replic precis perform trapp system give user fine grain control tradeoff precis perform cach store rang guarante bound current data valu store stale exact valu user suppli quantit precis constraint queri answer queri trapp system automat select combin local cach bound exact master data store remot deliv bound answer consist rang wider precis constraint guarante precis answer comput quickli paper defin architectur trapp replic system cover mechan cach data rang focus queri aggreg present optim algorithm answer queri precis constraint report perform experi demonstr fine grain control precis perform tradeoff offer trapp system interact data analysi control project datasplash se visu liz tion grow import tme se system tmome rger tm tmessi lef fpl sh syetoeus integr ted environ vig tingd tmre tingd queri visu repres tion demonstr te tmompon whi tmh ke fpl sh environmentx vig tion systemd dire tmtem nipul tion interf tme tmre ting modifi visu liz tionsd dire tmte nipul tion visu queri systemf overview databas system perform measur process speed quantiti data manag grown order magnitud recent year make increasingli sophist applic feasibl larger dataset databas queri languag chang difficult expert vast major databas user control continu output navig technolog refin dline control project berkelei develop technolog provid onlin behavior viqe visual interact queri 